Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Save 20~30 minutes on every shared memory run. #15

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

ezhang7423
Copy link

This addition to the code modifies the behavior of the multiprocessing.resource_tracker module in Python. It applies a monkey patch to ensure that the SharedMemory resource type is not tracked by the resource tracker.

The purpose of this modification is to prevent the training and validation language episodes from being removed from memory each time. By allowing these episodes to persist in memory, the code can save a significant amount of time, approximately 20 to 30 minutes, in each run.

The modification achieves this by overriding the register and unregister functions of the resource tracker. When a resource type is "shared_memory," these functions return None, effectively skipping the registration and unregistration process.

Additionally, the modification removes the "shared_memory" entry from the _CLEANUP_FUNCS dictionary of the resource tracker. This ensures that the cleanup function for shared memory is not called.

By making these changes, the code avoids the unnecessary overhead of reloading the training and validation language episodes, resulting in a considerable time-saving benefit during execution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant